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ABSTRACT 

University students’ practices of reading required course materials 
have rarely been studied systematically outside the laboratory and are 
given short shrift in course evaluation questionnaires. This study exam- 
ines, first, the reasons why a sample of instructors at a large university 
create selections of course readings and the factors, both pedagogical 
and otherwise, that influence their selections. Second, students’ patterns 
of reading in these instructors’ courses are studied to determine how 
they vary with the quantity of assigned readings, time of year, and the 
provision of systematic student feedback to instructors. Third, the most 
prevalent themes in student feedback about readings are explored, in tan- 
dem with instructors’ proposals about how to revise their reading selec- 
tions or teaching strategies. 
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RESUME 

L’utilisation de recueils de textes, requis dans les cours 
universitaires, a rarement fait Tobjet d’analyse sauf dans les laboratoires 
diriges par les psychologes. De plus, elle re 9 oit tres peu d’attention dans 
les questionnaires utilises pour I’evaluation des cours. Cette etude 
s’interesse d’abord aux causes incitant certains universitaires, oeuvrant 
dans de grandes institutions d’enseignement, a creer des recueils de 
textes ainsi qu’aux facteurs, pedagogiques et autres, qui influencent la 
selection des textes. Ensuite, les habitudes de lecture des etudiantes et 
des etudiants dans les cours selectionnes par cette etude sont analysees 
afin de determiner si elles varient en fonction du nombre d’articles a lire, 
du moment de Eannee scolaire et des commentaires fournis par les 
etudiantes et etudiants aux enseignants. Enfm, Particle explore les 
themes les plus importants contenus dans les commentaires des 
etudiantes et etudiants concernant les articles choisis par les enseignants 
ainsi que ceux des enseignants sur la maniere de revoir leur selection de 
textes et leurs strategies pedagogiques. 


Canadian university instructors commonly compile selections of 
journal articles and book chapters to supplement, or replace, course 
textbooks. For example, at my home institution of York University, 
which has 38,500 students and is Canada’s third largest university, the 
three leading suppliers estimated that they were asked to produce close 
to 1,250 different selections of readings for the 2000-2001 school year. 
Despite the popularity of reading kits, remarkably little is known about 
how students use them or how instructors can choose their contents 
most effectively. 

One source that a concerned instructor might consult is the substan- 
tial body of psychological research on university-level reading, in which 
experimental methods are used to understand the cognitive processes of a 
student who is engaged in a comprehension task in a laboratory setting 
(e.g., Alexander, Kulikowich, & Schulze, 1994; Beishuizen, Stoutjesdijk, 
& Van Putten, 1994; Marton & Saljo, 1976; Schnotz, Picard, & Hron, 
1993). For example, Schnotz et al. (1993) found that, when using a map 
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alongside text, successful learners distinguished themselves from their 
unsuccessful peers by focusing better on the information pertinent for 
constructing mental models. By tracking how students made their way 
through a hyptertext chapter, Beishuizen et al. (1994) determined that in 
contrast with their “deep processing” peers, students with “surface pro- 
cessing” learning styles moved chaotically among text cards and bene- 
fited from instructions about how information was organized. In a study 
of how students read physics materials, Alexander et al. (1994) propose 
that personally-involving details in a text (such as how Stephen 
Hawking’s research work may have slowed the progress of his disease) 
can seduce students’ attention away from key information. Such studies 
do provide insight into precisely how student readers with different 
learning styles formulate mental models, benefit from instructions, 
remain focused, or become distracted. But all of these studies — not to 
mention the numerous observations of readers’ eye movements (see 
Starr & Rayner 2001) — apply only to students who have their readings 
in hand. They do not address the thorny question of why university stu- 
dents decide whether or not to pick up a reading in the first place. 

A small but promising body of research addresses this issue. 
Chambers (1992), for example, reasons that students are deterred by uni- 
versity courses with high workloads. She recommends that instructors 
use a mathematical model for calculating the hours that various course 
components, including reading, require. Williams (1997) details how 
instructional strategies, such as pop quizzes, failed to encourage student 
reading in her tutorials. Based on a survey of her students, Williams rec- 
ommends strategies including interdependent learning, that could better 
foster reading. Mann (2000) advocates understanding more fully how 
students find their usually private and pleasant experiences of reading to 
be unsettled when they must read in order to be evaluated publicly in 
academe, while Francis and Hallam (2000) also discuss the emotional 
costs of students of eflForts to understand academic text. These studies 
extend the boundaries of reading research by considering the naturalistic 
contexts in which students learn, yet for the most part their recommenda- 
tions have not yet been tested systematically. 
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Course evaluation questionnaires, a potential source of systematic 
information for instructors about student reading, also fall short. For 
example, the standard course evaluation questionnaire used at York 
University, Canada, mentions reading only in the item; “Taking into 
account all aspects of the course (instruction, readings, assignments), 
how would you rate the course overall [on a five-point scale]?” Sample 
questionnaires provided in the course evaluation literature include at 
most two items on readings (Liow, Betts, & Lit, 1993; Marsh & Bailey, 
1993; Wilson, Lizzio, & Ramsden, 1997). Such items hardly provide the 
“specific, detailed, and idiosyncratic feedback” that Murray (1984, 
p. 123) deemed essential for formative purposes. 

In this investigation, I address limitations on existing research or 
evaluation questionnaires pertaining to student reading by examining 
empirically the contexts in which students read, including the instructors’ 
pedagogical aims, non-pedagogical influences on instructors, course 
workload, and timing in the academic year. 1 also provide a student ques- 
tionnaire that elicits systematic and detailed feedback on course readings, 
and assess the efficacy of providing instructors with this feedback. 

The first research question addressed is why instructors decide to 
assign reading kits and how they choose their contents. The accounts of 
six instructors help us to understand how pedagogical aims influence kit 
development. Surprisingly, their accounts show that factors unrelated to 
pedagogical aims often influence how readings are chosen. 

Next, 1 explore how students’ reading patterns vary by other aspects 
of course context. Prompted by arguments that a high workload deters 
learning (Chambers, 1992; Franz, Ferreira, Loh, Pendergast, Service, 
Stormont, Taylor, Thambiratname, & Williamson, 1996; Gibbs, 1992, 
cited in Flartley, 1998), 1 use student questionnaires to examine whether 
students’ claims to recall readings are affected by the quantity of read- 
ings an instructor assigns and whether reported recollection varies by the 
point in the academic year at which a reading was assigned. I also 
explore whether receiving systematic feedback from students leads 
instructors to change their courses in ways that increase students’ recall 
of readings. The complex methodological issues that render these explo- 
rations tentative are taken up in some detail. 
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Finally, 1 determine students’ most prevalent concerns about reading 
material and show that these vary little amongst the disciplines studied. 
Students typically recommend that there be less required reading, and 
that it be easier, better written, and more relevant. The instructors did not 
take these recommendations at face value but, rather, reframed them in 
ways consistent with their pedagogical aims. The new instructional 
strategies that the instructors formulated are discussed with reference to 
studies on reading and learning processes (e.g., Alexander et al., 1994; 
Baxter Magolda, 1992; Bean, 1996; Schnotz et al., 1993) and on instruc- 
tors’ and students’ differing conceptions of learning (e.g.. Child & 
Williams, 1996; Franz etal., 1996; Liowetal., 1993). 

METHODS 


First Year: 1998-1999 

In 1998-1999, a sample of six instructors from a large urban univer- 
sity where reading kits are frequently used, agreed to participate in this 
study. Although the sample was opportunistic, the instructors and the 
eight courses they were teaching with kits, covered a wide range of pos- 
sibilities. The instructors came from disciplines in the general areas of 
business, humanities, and the social sciences. (For ethical reasons, the 
names of these general areas, capitalized and set in quotation marks, are 
used as pseudonyms for departments. Pseudonyms are also used for the 
names of authors of course readings.) Half of the instructors were offer- 
ing their course at this institution for the first time. One of the courses 
was at the 2nd year level, five were third year courses, and two were 
fourth year courses. In two courses, the reading kit was the only required 
reading material. In five others, textbooks were used; in one, several 
novels were assigned. 

First interview with instructors. Each instructor participated in a 
one to two hour long structured qualitative interview asking them to 
describe their course(s), their reasons for using a reading kit, and their 
method of selecting readings. Probes included whether materials from an 
earlier kit, colleague or canon were chosen; whether (and how) student 
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feedback was sought; and whether criteria other than the content of a 
reading, such as its publication date or its author’s sex, were considered. 

First student survey. Questionnaires were administered to 183 stu- 
dents in the eight courses between March 24 and April 9, 1999 (i.e., the 
second- or third-last week of term). The questionnaire first listed the 
author and title of each kit reading. Students were asked to indicate 
whether it should be kept or dropped the next time the course was 
offered, whether they had no opinion of it, or whether they did not 
remember it. Qualitative comments about each reading were also 
invited. The questionnaire concluded with a pair of open questions, ask- 
ing what distinguished kit materials that should be kept, and what other 
changes to course materials were recommended. These questionnaires 
required 10 to 25 minutes to administer. Confidentiality was protected 
by having instructors leave the room while questionnaires were adminis- 
tered, by asking students not to identify themselves, and by providing 
typed feedback to instructors only after final grades had been submitted. 

The proportion of registered students who participated in the study 
varied greatly from course to course, from approximately 25% to 
approximately 68%.' These modest rates can best be explained by low 
attendance late in the winter since, in all but one class, most students 
present appeared to be participating. 

Questionnaire results were provided to instructors in reports that 
included: a table showing for each reading, the number of students who 
said they would keep it or drop it, had no opinion of it, or could not 
remember it; a list of the quartiles of readings (or as close to quartiles as 
possible) that had received the highest and lowest ratings;^ a list of the 
readings that at least 25% of students reported not remembering; a ver- 
batim transcript of responses to the open-ended questionnaire items; and 
a summary of themes in those responses. 

Follow-up interview with instructors. These reports were dis- 
cussed in a second interview with each instructor. Participants were 
asked to think aloud about their reactions to the report, identify patterns 
in students’ ratings, and indicate how they might use the report. 
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Second Year: 1999-2000 

Follow-up student survey. In 1999-2000, five of the eight courses 
studied in 1998-1999 were again offered by four instructors from 
“Humanities” and “Social Science” disciplines. Survey questionnaires 
based on the updated kit materials were completed by 1 5 1 students in 
these courses between March 20 and March 23, 2000 (i.e., the second- or 
third-last week of term). Procedures were unchanged, except that in one 
course where time was short, students were asked to return completed 
questionnaires at the subsequent lecture. Because only two of these 25 
students complied, this course is omitted from analysis. In the remaining 
courses, between approximately 27% and 80% of registered students 
completed questionnaires. The results were compiled following the same 
procedures as in 1998-99, with the addition of tables comparing the 
1998-99 and 1999-2000 distributions of ratings and percentages of 
remembered materials. 

Final interview with instructors. These interviews addressed: the 
changes in students’ evaluations of readings between year 1 and year 11; 
instructor’s predictions about which readings would obtain the highest 
and lowest scores; patterns in the readings that actually had obtained the 
highest and lowest scores from students; predictions about the times in 
the academic year for which students do the most reading; a chart show- 
ing how reported recall changes over the year; and the utility of the sys- 
tematic feedback 

Sampling Issues 

While the samples of students and readings studied here are large — 
334 and 394 respectively — the research involves only six instructors and 
12 courses, chosen using non-random methods. Some parts of the analy- 
sis are limited to the four courses in which students completed question- 
naires in both 1998-99 and 1999-2000. Therefore, the qualitative 
analysis of instructors’ statements must be seen as a tentative sketch that 
outlines tendencies of the sample rather than making claims about a pop- 
ulation. I signal in the text that the results hold for “the sample of instruc- 
tors” (rather than “instructors”). The quotations selected illustrate themes 
mentioned by more than one instructor, unless 1 have noted otherwise. 
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In the quantitative analyses (in which courses are the units of analy- 
sis), I have used simple graphs and tables, rather than the formal statisti- 
cal methods that require random samples. Further, as they arise, I 
identify interpretation issues that result from the small sample or for 
other methodological reasons. 


RESULTS 


Choosing Readings 

Instructors’ reasons for using kits. Despite their varied disciplines, 
the sample of instructors had considerable common ground in the ratio- 
nale for using reading kits. Most mentioned that kits could be used more 
effectively than textbooks to challenge students, echoing one of the criti- 
cisms of textbooks in Dorn (1989). A “Business” instructor said; “After 
the first two levels of courses in this area, students have 80 to 90 credits, 
and they can go beyond the textbook by starting to criticize topics and 
the discipline.” A “Social Science” instructor remarked that “[textbook] 
authors think that the students are cretins, that they know nothing. They 
have figures, they choose an important author, say, Alvarez, and have 
two pages on him.” “1 want to show them a killer,” was a “Humanities” 
instructor’s comment on his most difficult reading. 

Another frequently-mentioned reason for using kits was that they 
expose students to diverse sources. One “Humanities” instructor 
explained, “1 wanted to have some examples of art, so 1 wanted some 
poetry and short stories. 1 had a number of primary texts in philosophy 
— brief ones, it was a kind of tokenism. 1 wanted to have examples of 
contemporary academic articles.” A “Social Scientist” said, “1 have a kit 
because there are no good Canadian textbooks and therefore I have to 
supplement the American text with Canadian material. This reason 
might not have been as prevalent had 1 included Natural Sciences 
instructors in the study, because their disciplines employ less diverse 
paradigms and have less contested canons (Schachter, Christenfeld, 
Ravina, & Biloux, 1991; Smeby, 1996). Other reasons for using kits, 
which the sample of instructors mentioned less often, were that no suit- 
able text existed, that kits could cover a better range of publication dates. 
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that kits best reflected instructors’ politics or learning traditions, and that 
kits were less expensive for students to purchase. 

Although the instructors concurred that students should be exposed 
to diverse sources, they disagreed about whether readings by authors of 
diverse identities should be sought. For example, to half of this small 
sample, an author’s sex mattered. “I’m aware that 50% of the students 
are women,” noted a “Humanities” instructor, while one “Social 
Science” instructor indicated that she had discussed with students the 
shortage of female authors in her field. Their approaches to reading 
selection reflect feminist and other pedagogues’ arguments for the 
recognition of diversity in the curriculum (e.g., Boud, 1993; Thomas, 
1990), though they do not necessarily problematize concepts such as 
“integration” or “providing role models” (Andersen, 1988; Fisher, 1988). 
In contrast, other instructors claimed that an author’s sex was of no 
import. Interestingly, these instructors overestimated the percentage of 
female authors in their kits. For example, a “Business” instructor, who 
guessed that 30% to 40% of his readings had female authors because 
“there are many women accountants,” found only 22% actually did. A 
“Social Science” instructor, who believed that over half her readings had 
female first authors, found the correct percentage to be 41%. 

A private underiaking. At Canadian institutions, instructors typi- 
cally have complete autonomy in their selections of reading kit materi- 
als. Four of the six instructors interviewed depicted the choice of their 
kit materials as a remarkably solitary task. Apparently in near silence 
about their purposes, they selected new materials by working through 
the library stacks, eyeing others’ syllabi, assessing new textbooks and 
journals, and attending professional meetings or courses related to 
course topics. They explained their isolation variously by referring to 
confidence about reading choices, lack of relevant colleagues in a small 
field, and habits formed while working as an ostracized sessional 
instructor or as an untenured faculty member fearful of scrutiny. Some 
were surprised to realise Just how isolated their work had been, as in 
these examples: “No [I haven’t spoken with anyone]. Not because I 
don’t want to. Nobody. 1 never had a conversation with colleagues about 
readings. It is seldom that [instructors in my discipline] raise questions 
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like this,” and “I never talked to a living soul about my kits! My col- 
leagues now won’t pull me down, the students appreciate my work and 
my colleagues know it. So why won’t I speak about kits?!” 

The silence about kits may not be golden. None of the instructors 
consulted the professional librarians who select this university’s library 
collections and who are well informed about new research and resources 
in their fields. Such consultation with librarians or colleagues might 
have averted certain errors instructors felt they had made when choosing 
materials. For example, one instructor who had taught her course several 
times confessed, “Sometimes when I’m pressed for time and choose an 
article based on the abstract and 1 haven’t read it, then when I get to it I 
think, ‘holy shit!’ ” An instructor preparing a new course explained, “I 
had less than a month to make the selection. I based [it] on what 1 knew, 
or when 1 thought, ‘this is a nice looking article,’ after reading the 
abstract and the first two or three pages. I try to put a brave face on when 
they say, ‘this is a horrible article.’ ” 

Student feedback quandaries. The teaching evaluation question- 
naires students routinely complete provide little information about stu- 
dents’ views of course readings. Thus, the instructors 1 interviewed 
tended to seek students’ impressions of the readings by other means, e.g., 
“1 get ad hoc feedback; I might say, ‘I found it blah blah blah’ and they’ll 
tell me they agree. But it’s not at all systematic” or “I get reactions in 
class and in their journals. I should evaluate, but here we don’t and 
there’s no point in sticking out too much. [In another course] if they’ve 
not read an article well, they’re quite quiet about it.” The following quo- 
tation describes one of the most thorough attempts to obtain feedback: 

In October 1 asked students for feedback, and I wanted it writ- 
ten, because oral feedback might intimidate students. If they 
say ‘sorry sir, it’s bad’, under those conditions, it is a coura- 
geous student. 1 asked their opinion of the lectures, the topics, 
the workload, the tutorials, the readings, and any recommen- 
dations for improving the course. Students said the readings 
were okay. 

The instructors criticized these methods because they lacked rigor or were 
untimely, and because of the difficulty of preserving student confidentiality. 
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Further, one instructor was particularly concerned about the conflict 
between her pedagogical values and the values implicit in students’ feed- 
back. Faced with poor formal evaluations in a course she believed had 
gone well, she commented; 

For me, [students] have to be really independent and most of 
them are dependent. It’s true that they are independent in that 
they work outside, but they are really consumers. If they are 
unhappy with the course, they say, ‘but 1 paid for it,’ and not, 

‘1 want to work in the area of [X] and this course is not 
related to my objectives.’ 1 wanted a course where students 
would feel able to speak. I thought 1 had done that but it 
seems it is not the case. But I am stubborn. Even if I am 
untenured, I will do what I think best to do and that’s it. 

This instructor’s perception that feedback can be embattling, particu- 
larly when it is employed in tenure decisions, illustrates Murray’s (1984) 
point that evaluation practices may lower standards or reduce morale, 
depending on how the evaluation outcomes are used. While Murray 
(1997) later concluded that student evaluations actually have positive 
effects on several aspects of teaching and that “there is no clear evidence 
that student evaluation of teaching has led to negative side-effects com- 
monly attributed to it” (p.21), the faculty survey data he reviewed show 
substantial variation among institutions in instructors’ perceptions about 
negative effects. 

Using Readings: Course Context and Student Reading Patterns 

We now turn from the pedagogical and non-pedagogical factors that 
influence instructors’ decisions about kits, to the questions of how much 
and when students read. This section explores, first, whether students’ 
claims to remember readings are associated with the quantity of assigned 
readings; second, whether students change their reading patterns in pre- 
dictable ways during the academic year; and third, whether students’ 
reading patterns changed between 1998-99 and 1999-2000 (owing 
either to the research intervention of providing systematic student feed- 
back or to other intervening factors). 
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The workload context. In their qualitative recommendations, at 
least 10% of students in all but one course requested that instructors thin 
kit material, by using fewer or shorter readings. Typical comments 
include: “Shorter readings are better just because it is easier to stay 
attentive until the end;” “Too much material to take in all at one time 
sometimes. Too much material, needed to spread out so can be better 
understood,” and “Overall, I think the readings were good, but too long 
— I think a summarized version would be easier for the students to 
understand.”Do students read more in courses where a smaller number 
of readings are assigned? The percentage of readings students reported 
remembering in each course is plotted against the number of assigned 
readings in Figure 1. No relation appears: regardless of the quantity of 
readings required, students in each course said they remembered some 
70% to 90% of readings. Accordingly, the higher the expectations these 
instructors set, the greater the absolute number of readings their students 
say they remember (see Figure 2). This finding, at first glance, appears 
to overturn hypotheses that a high course workload deters from learning. 

However, this conclusion may be too hasty. The measure used here, 
though suited to a quick one-time-only in-class survey, is crude. 
Reports of whether an article is remembered may be incorrect, remem- 
bering an article is not necessarily the same as having read it, and even 
reading an article does not guarantee learning deeply, as opposed to 
“skim[ming] along the ‘surface’ of things” (Chambers, 1992, p. 145). 
Further, the measure accounts for neither the number of pages in a typi- 
cal reading in a discipline, nor the additional texts or novels set in most 
courses. Additional research that accounts for these factors is needed 
for the relations between workload, reading, and learning to be exam- 
ined more effectively. 

The timing of readings. The six instructors’ decisions about read- 
ing requirements are also informed by their beliefs about how students’ 
reading ebbs and flows with the academic year. For example, one 
instructor from “Social Science” course believed that students would 
read most late in the fall term, while a “Humanities” instructor expected 
students to read least in November and March. 
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Figure 1 

Percentage of Readings Recalled, by Number Assigned 
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I divided the readings for each course into six groups of roughly 
equal size, according to where the reading fell on the list of kit materials 
for the year/ The first sixth corresponds roughly to readings set for 
September; the next sixth, to October, and so forth. The grouping is 
rough because instructors might drop readings during the year, change 
their order, or assign heavier reading loads at times when they believe 
students read more. 

The mean percentages of students in each course who said they 
remembered readings from each ‘month’ are shown in Table 1. Variation 
among the courses appears much more marked than any time-linked ten- 
dency. The exception is that students in full-year courses appear to report 
remembering fewer readings toward the end of the year. This may be a 
methodological artifact, as student questionnaires were administered 
before the March readings had been covered completely. Even so, the 
“Social Science A” instructor argued that there appeared to be a genuine 
drop in recall toward the end of the year: 

They had two assignments in February and March, based on 
readings, so it’s odd to see they say they’re not reading them, 
even though they had to use all relevant course materials for 
the assignments, which were given before you came to do the 
survey in March. And most used them, too. 

When the sample of instructors learned what students reported 
remembering of their own course readings, they began to question their 
usual methods of assigning readings. For example, two instructors whose 
students’ relatively poor recall patterns are shown in Figure 3 said: 

Probably 1 should pile it on at the beginning. It slipped out of 
my consciousness, the crowding [of readings assigned at the 
end of the year]. It suggests 1 should average out the readings, 
look at the number of pages per week, and figure out what I 
can do. 1 will work on being more scientific about the read- 
ings and weight them toward the beginning of the course. 
(“Humanities B”) 
and 

Maybe I’m too keen or too soft. What do the others do? 
Assign less? Or what do they do to make students read the 
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Table 1 

Mean Percentages of Students who can Recall Reading, by Approximate Time of Year that Reading was Assigned, 
Year and Course 


1998-1999 

September 

October 

November 

January 

February 

March 

'’students 

’’readings 

“Business A” 

- 

- 

- 

86.2 

81.5 

82.7 

27 

20 

“Business B” 

- 

- 

- 

84.8 

90.3 

89.2 

15 

26 

“Humanities A” 

71.3 

73.8 

69.8 

66.3 

61.3 

58.3 

16 

32 

“Humanities B” 

82.1 

73.8 

75.3 

59.2 

76.2 

60.7 

56 

36 

“Social Science A” 

83.2 

67.4 

79.5 

88.4 

86.3 

82.6 

23 

39 

“Social Science B” 

84.2 

72.9 

86.5 

77.0 

83.5 

69.2 

19 

44 

“Social Science C” 

82.0 

91.2 

89.5 

96.5 

84.2 

78.9 

19 

39 

1999-2000 

“Humanities A” 

79.0 

74.4 

79.0 

72.8 

72.5 

62.2 

20 

57 

“Humanities B” 

91.4 

84.4 

86.7 

82.6 

70.0 

50.0 

72 

27 

“Social Science A” 

85.1 

81.5 

71.8 

90.3 

89.2 

72.4 

39 

28 

“Social Science B” 

94.3 

91.7 

87.1 

86.7 

92.1 

94.2 

20 

39 

Unweighted average 

of all courses 

83.6 

79.0 

80.6 

81.0 

80.7 

72.8 

326 

387 


Note: “Business A” and “Business B” were winter term courses, not full-year courses. 
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Figure 3 

Highlights of Recall Trends 
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stuff? Your graph is very interesting because it forces us to ask 
other questions, for example, should we give assignments on 
the readings? In [my discipline], when we give assignments 
they are supposed to go to the library. (“Humanities A”) 

In contrast, the “Social Science B” instructor, whose 1999-2000 students 
reported remembering the highest proportion of readings (see Figure 3), 
thought a reason might be: 

The organization of the course. We discuss the readings, stu- 
dents have to do presentations on the readings, it’s a compul- 
sory thing. They have to read three readings at least [for the 
presentations], it’s not much, but.. .And also I think the fact 
that 1 ask questions about what they understand and what 
they don’t, and then I answer them in the next class, quoting 
the authors. 

1998-99 vs 1999-2000. Student recall of the readings was greater 
in 1999-2000 than in 1998-99 for each of the four classes surveyed 
(Table 2). The most dramatic improvements in reports of remembering 
readings occurred in “Social Science B:” 64% of readings were said to 
be remembered by at least three quarters of student respondents in 

1998- 99, while in 1999-2000, this held for 95% of readings. The 
improvements in other courses ranged from 12 to 19%. 

These changes were accompanied by substantial increases in students’ 
ratings of readings they recalled. Students who said they remembered a 
given reading had indicated whether it should be kept (coded 2), or 
dropped (coded 0), or whether they had no opinion on the matter (coded 
I). Using the 1998-99 student data, I grouped readings in each course 
according to whether they fell in the top quartile, bottom quartile, or 
middle of the course’s ratings."* In Table 3, we see that when the 1998-99 
cut-off points for the quartiles in each course were applied in 

1999- 2000, ratings had improved. In “Humanities A,” “Social Science 
A,’’ and “Social Science B,” the percentages of readings with high scores 
rose by 24, 10, and 29, respectively. In “Humanities B,” the improve- 
ment occurred in the category for moderate scores. 

Fact or artifact? Although the changes between 1998-99 and 
1999-2000 are impressive, they should not be attributed too quickly to 
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Table 3 

Distributions of Students’ Assessments of Readings They Recalled, Before and After the Research Intervention, by Course 


“Humanities A” “Humanities B” “Social Science A” “Social Science B” 

Mean Score 1998-99 1999-00 1998-99 1999-00 1998-99 1999-00 1998-99 1999-00 


High 

25% 

49% 

33% 

30% 

26% 

36% 

25% 

54% 

Medium 

44 

42 

25 

44 

51 

46 

61 

33 

Low 

31 

_9 

42 

26 

21 

18 

14 

11 


100 

100 

100 

100 

100 

101 

100 

100 

’’readings 

(32) 

(57) 

(36) 

(27) 

(39) 

(28) 

(44) 

(39) 


Note: The cutoff points for high and medium scores, respectively, were; 1 .8 and 1 .4 in “Humanities A;’’ 1 .6 and 1 .4 in 
“Humanities B;” 1.8 and 1.3 in “Social Science A;” and 1.6 and 1.2 in “Social Science B.” 
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my intervention of providing instructors with systematic student feed- 
back. In this section I address briefly two methods artifacts that could 
also explain the improved ratings. First, the students participating in this 
study were self-selected, rather than selected randomly. This means that 
changes in how students selected courses between 1998-99 and 
1999-2000 could contribute to the improved recall and assessment of 
readings in 1999-2000. For example, “Humanities A,” “Humanities B,” 
and “Social Science B” were all taught for the first time by their instruc- 
tors in 1998-99. The 1999-2000 students might have had more informa- 
tion about the courses and their instructors, and therefore could have 
matched their course selection to their interests and aptitudes better than 
the 1998-99 students. Nonetheless, ratings also improved in “Social 
Science A,” which had been offered repeatedly by the same instructor. 

Second, we must consider whether the improved ratings could result 
from refinements the instructors ordinarily might have made to their 
courses, independently of the influence of the systematic feedback. The 
“Humanities A,” “Humanities B,” and “Social Science A” instructors 
thought that they had already finished revising their kits before receiving 
the systematic feedback. The “Humanities B” instructor made a few 
more changes after seeing the systematic feedback, but the other two 
instructors made no further changes and still had higher ratings. 

This suggests that instructors’ ordinary methods of revising course 
materials, absent of systematic feedback, can sometimes be very success- 
ful. However, as we shall see later, at other times, instructors can make 
spectacularly poor predictions of students’ preferences. Finally, we should 
consider that the two instructors whose ordinary revision plans had con- 
curred with students’ recommendations said that participating in the initial 
interview about their readings had been constructive nonetheless: 

Instructor: 1 had a better selection of readings because of this 
experience with you, because 1 tried to have 50% by men and 
50% by women. 

Researcher: Why are you taking gender so seriously — it was 
only a question 1 asked you last time [i.e., not a directive]? 

Instructor: Your background influences the way you look at 
some issues. 1 found it was an interesting challenge. 
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Instructor: 1 think our focused discussion of the course kit at 
the end of the year really made me rethink a lot. In any case, 1 
believe in the value of more systematic evaluation. 1 did it 
intuitively and ad hoc. 

Researcher: Why do you believe in it, if the results are the 
same as what you did yourself? 

Instructor: One reason to do a systematic assessment is so 
they get a sense their feedback matters, because 1 don’t 
always tell them that I’m considering their feedback. 

Since collegial discourse about reading kits is rare among the instructors 
1 studied, participating in the first research interview gave some instruc- 
tors an unusual opportunity to reflect on their pedagogy. Accordingly, 
the research intervention should be considered to consist of both the ini- 
tial interview, which invites reflection, and the provision of systematic 
feedback. This suggestion is consistent with statements that feedback has 
heightened value when it is accompanied by self-assessment (Brinko, 
1993) or consultation (Murray, 1984). 

Student Feedback and New Instructional Strategies 

Surprises. Brinko (1993) states that feedback is most effective when 
it creates cognitive dissonance. Certainly most instructors in the sample 
seemed to find the surprising student feedback to be the most informa- 
tive, as the following quotations indicate: 

First interview: 1 say to the students, you have to fly casual 
[with the reading by Hecht], we know each other. Colleagues 
would say ‘the kids can’t handle it.’ I would say, they’re not 
‘kids’ and yes, they can, look at the course evaluations. 

Follow-up interview, after reading student comments such as 
‘Hecht wai pretty impossible to understand. I didn ’t get any- 
thing out of the reading’: For Hecht, there are massive com- 
ments. 1 say, ok, that could make me drop it. 
and 
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[1 expect high ratings for] Anshaw, because students could 
relate to the experience of these women. I had two mature stu- 
dents who said this was the experience of their parents. 
Divakaruni and Gilman — again, 1 remember one student 
who said ‘this is the experience my mother had.’ [Instructor 
learns that none of these three are among the most positively- 
rated readings and that only a quarter of his predictions 
about positively-rated readings are correct.] I’m surprised. I 
don’t know what to say. 

To determine how attuned the instructors were to student preferences, 1 
asked the four instructors whose students had been surveyed in 
1999-2000 to predict which readings students would rate among the best 
and worst. Despite his accurate predictions about 1998-99, the 
“Humanities A” instructor was correct only 1 1 out of 28 times. Another 
instructor, who confidently placed half of her students’ favourite read- 
ings in the worst-rated category, had 1 1 correct predictions out of 23; a 
third instructor had just four correct predictions out of 12. The instructor 
who fared best, with 14 correct predictions out of 20, was dubious about 
her predictions: “I’m just doing it to get it done, 1 really don’t know.” 
Therefore, although students rated kit materials more favourably in the 
study’s second year, this sample of instructors either had not become 
conversant with student preferences, or had become fairly conversant 
without realizing it. 

From surprises to guidelines. Many of the instructors’ conclusions 
about the systematic student feedback were discipline-specific. For 
instance, “Business” students’ enthusiasm for a reading issued by the 
Office of the Auditor General might not carry over to other pro- 
grammes. However, in other regards, students made strikingly consis- 
tent recommendations. 

This conclusion is based on the qualitative feedback students gave 
when asked how they identified readings that should be retained and 
what other changes to the kit they would recommend. Averaging across 
courses, 78.0% of students provided such recommendations. Table 4 
shows the distribution across courses of major themes (mentioned by at 
least 30% of students in a course), moderate themes (mentioned by 
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Table 4 

Distribution Among Courses of Top Themes in Qualitative Student 
Comments 


Number of courses where theme is: 


Theme 

Maior 

Moderate 

Absent/m inor 

Less 

1 

10 

1 

Easier 

6 

6 

0 

More challenging 

0 

3 

9 

Better written 

5 

7 

0 

Relevance 

7 

5 

0 

Discipline-specific 

2 

7 

3 

Other 

0 

2 

10 


10-29% of students), and minor themes (mentioned by 0-9% of stu- 
dents). As discussed earlier, several students say that they would prefer 
to have less reading material. However, they were even more concerned 
that readings be easier, better-written, and ‘relevant.’ In several courses, 
students also made discipline-specific recommendations. 

In responding to this feedback and the lists of most positively- and 
negatively-rated readings, the instructors first assessed whether they had 
received any information during the year that would counter-indicate the 
feedback. For example, one instructor questioned students’ poor ratings 
of a reading that many had used effectively in their exams. Another 
instructor disagreed with students’ high ratings of materials that had, in 
her view, left them ‘completely lost for over four weeks.’ The instructors 
then tried to identify which of their teaching strategies had led to the 
feedback and to think of ways to change these strategies in response to 
the feedback. 

In the following sections on students’ main themes, I will show how 
the instructors strove for consistency with their pedagogical goals of 
challenging students, reinforcing central course objectives, and exposing 
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students to diverse sources. At times, this process made salient differ- 
ences between students’ and instructors’ orientations to learning — the 
former, characterized variously in the literature as mercenary, con- 
sumerist, literal, or practical, and the latter, depicted as passionate, learn- 
ing-oriented, holistic, or analytical (Cruse, 1987; Child & Williams, 
1996; Franz et al., 1996; Liow et al. 1993). 

‘Shorter, easier, and better written.’ Although students’ concerns 
about the amount of reading have been addressed, they are worth revisit- 
ing to see how length, difficulty, and poor writing compound one 
another. Students’ remarks included: “For me, 1 found the reading very 
hard to understand and with the length of the articles 1 was not able to 
finish the majority of them;” “Too lengthy becomes boring and tedious, 
because most of the time, people are just repeating one idea over and 
over;” and “The other course material is just too monotonous and diffi- 
cult to understand.” 

These statements are corroborated by reading research: Hartley and 
Trueman (1992) find that many psychology journal articles are better 
suited to graduate readers than undergraduates, when their reading ease 
is systematically estimated. Bean (1996) remarks that students are ‘baf- 
fled by the strangeness and complexity of primary sources and by their 
unfamiliarity with academic discourse’ (p.l33). 

The instructors I studied formulated several approaches to these inter- 
related concerns. As a last resort, they suggested dropping a reading — 
“Fielding is brilliant but the students have been complaining about it for 
five years. They don’t get it. It’s finally out. Seth is poorly written and 1 
decided just to can it.” Several instructors who endorsed this emphasized 
that they were not ‘catering’ to students. A less drastic approach was to 
make the rationale for assigning a reading more explicit: 

In a second year course, students should be reading articles, 
identifying the thesis, looking at the author’s approach, but 
maybe the course still does not give them enough on how to 
do that. For example, there’s a reading on human sacrifice in 
which the author is trying to explain how religious world 
views lead to sacrifice, and it is only at the end that he 
describes sacrifices. 1 pointed out that this was a rhetorical 
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decision. They should be challenged with difficult readings 
but something that comes through is that, if it is difficult 
because they’re left on their own to judge it or if it is analyzed 
but insufficiently, that’s bad. 

Unknowingly, these instructors were recommending strategies con- 
sistent with reading research findings. For example, Schnotz et al. (1993) 
found that successful learners in a laboratory exercise were distinguished, 
not by the quantity of information they could command, but by their abil- 
ity to determine which information could best be used to construct mental 
models. In practice. Bean (1996) suggests that students be taught how 
experts read — that is, how they vary their reading speed for different 
parts of an article, how they adjust to primary and secondary sources, 
how they identify the rhetorical function of text sections, and so forth. 

‘Readings should be relevant.’ Students invoked the concept of 
‘relevance’ in two main ways. Often, students regarded a reading as rel- 
evant if it was taken up in class or if its concepts were reinforced in 
course requirements, as these quotations suggest: “Some were irrelevant 
articles. If the readings are not mandatory do not include them in the 
course kit and expect us to purchase it. Last semester’s kit cost $65 and 
only some of these readings were compulsory;” “It became obvious in 
tutorials which readings seemed valuable to the overall course. Noticing 
the amount of discussion on each of the readings showed that some 
were more important to the course;” and “1 based my opinions on the 
assignments that were given out. 1 think that a lot of the readings were 
just other information that was not useful for assignments and exams.” 
The students’ emphasis on this type of relevance in their qualitative 
comments mirrors Liow et al. (1993) conclusion that students regard 
exam preparation as a more important course goal than do instructors, 
as well as Williams’ (1997) finding that many students feel they would 
prepare better for tutorials if their preparatory reading work were 
graded in some way. 

The students’ comments are also confirmed by their quantitative rat- 
ings of course materials. Instructors discovered that some surprisingly pos- 
itive or negative ratings could derive from how well they had explicated 
the readings in lectures or required their use in assignments or exams:^ 
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Maybe that’s why they recall the last three, they’re all used in 
an assignment. That’s true of all the other most positively rated 
ones except Choy, Roberts, and Woodrow. Woodrow is a 
really interesting discussion of meat-packing, and 1 really get 
into it in class, these carcasses coming along the assembly line, 
and 

[1 did not expect students to rate the case studies so posi- 
tively] because the cases are much more demanding, and 
there’s more pressure in class to discuss them. But I give them 
the 12 in class and I tell them that one of them is going to be 
on the final so usually they study them quite well. 

Relevance: Take Two. When requesting ‘relevant’ readings, students 
also voiced strong preferences for the familiar, the contemporary, and the 
personally meaningful, such as; “Any current events might be included 
during the time they are happening, especially if there is a relation to the 
[disciplinary] context;” “Some are very outdated. It is hard to discuss in 
1999 about ideas as old as 1992;” and “Materials added 1 would like to 
see would be articles that are less philosophical, and that pertain more to 
society, controversies, issues that are going on in our lives.” 

These instructors applauded students’ interest in having academic 
work speak to their experiences — cases in point are that a “Humanities” 
instructor was pleased to see how readings about working-class women 
resonated with some students, and that a “Business” instructor aspired to 
design courses that students could apply at work. Nonetheless, several 
instructors insisted that students should develop the analytic skills to deal 
with both the personal and the general, both the familiar and the strange: 

What they like is related to, except for some surprises, the 
exams or things that are easy, or experiential work, which is 
good. At the same time, they have to go beyond their experi- 
ences, to think outside them in a certain sense. The theory is 
not something we can forget if we are studying at the univer- 
sity level and it is a framework that helps to organize the way 
we see the world. 

and 
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[On a negatively-rated reading about McDonald’s restaurants] 

Some students working at McDonald’s had a happy experience 
there and they disagreed with [the author, who criticizes 
McDonald’s], People in the class said, ‘but my experience is x’ 
and I tried to make students aware that to reject a reading 
because of its method of using one person’s experience, and yet 
to base your analysis on your own experience, is a bad analysis. 

In response, one instructor speculated that students’ interest in contem- 
porary substantive topics could be an avenue for introducing difficult 
concepts or methods: “I’m sobered by the universal voice that the mater- 
ial is abstract. If 1 were clever, I would use the desire for the contempo- 
rary and feed it back into the course. For example, [by analyzing] ads 
that use the Adam and Eve myth.” 

These instructors’ ambivalence about students’ desire for ‘relevance’ 
has its counterpart in the pedagogical literature. One group of 
researchers locates students’ personal experiences at the heart of learn- 
ing. Boud (1993) positions experience as the basis of knowledge; Franz 
et al. (1996) proposes that learning is the process of finding personal 
meaning in material; and Baxter Magolda (1992) illustrates how matur- 
ing undergraduates increasingly see themselves (rather than authorities) 
as the source of knowledge. In the classroom, Hollander (2000) and 
Widerberg (1998) are among the many who use personal journals or sto- 
ries as teaching tools. Other researchers decry students’ penchant for the 
personal. Alexander et al. (1994) warn that ‘seductive details’ and per- 
sonally involving content can detract from understanding material that 
should fascinate students on its own merits. Bean (1996), who refers to 
‘the ‘cognitive egocentrism’ of new college students who have trouble 
walking in the shoes of persons with unfamiliar views and values’ 
(p. 135), considers this part and parcel of a normal developmental stage 
that can be fostered by effective pedagogical strategies. 

Finally, students’ comments about personal relevance and their disci- 
pline-specific recommendations at times mentioned identity politics 
issues. Examples include requests that the Bible not be used because it 
does not reflect all students’ faiths, that a person of colour delete material 
on race, that a Quebecois instructor place less emphasis on information 
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about Quebec, and that a feminist change her course readings because ‘a 
lot of the femininity material isn’t as important’. Cumulatively, these 
comments indicate that students can regard readings as statements about 
the bodies or identities in the classroom, with the potential to feel or to 
create a chilly climate. Some of the comments also illustrate Magnusson’s 
(1999) point that students can use teaching evaluations to resist curricula 
that address systemic inequities, with implications for tenure decisions. 

CONCLUSION 

When choosing reading materials, the instructors in this sample from 
five disciplines were informed by pedagogical objectives, such as chal- 
lenging students, exposing students to diverse sources, selecting material 
at a suitable level of reading difficulty, and reinforcing central course 
themes. Less obviously, their choices were informed by disparate non- 
pedagogical factors, including the desire to reduce students’ expenses, 
isolation from other instructors with different statuses, and dissatisfac- 
tion with the format or implicit meaning of student evaluation methods. 
Another factor — the use of sex (or other author identities) — calls into 
question the very distinction between pedagogical and non-pedagogical 
factors, as this sample of instructors came to no consensus as to whether 
authors’ identities influenced learning. 

Little is known about university student reading patterns outside the 
laboratory context. In this study in naturalistic classroom contexts, I 
examined how recall of readings is affected by the quantity of readings 
required for a course and by timing in the academic year. Students’ 
recall of readings was found to be unaffected by workload: regardless 
of whether a course required 7 or 57 readings, or quantities in between, 
students remembered roughly the same percentage of readings. This 
finding appears to overturn Chambers’ (1992) and others’ proposals 
that an excessive workload can detract from learning, though further 
study is indicated because of the cude measures that were employed. 
Further, patterns of student recall over time vary greatly from course to 
course, but show a possible decline for readings assigned late in the 
academic year. 
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Students’ reported remembering and ratings of course materials 
improved markedly from 1998-1999 to 1999-2000. Some of the improve- 
ment might be attributed to a research intervention in which I provided 
instructors with systematic feedback from students. Alternately, the 
improvement might be attributed to the revisions instructors ordinarily 
make to their kits in the absence of systematic feedback. Yet, the instruc- 
tors tended to indicate that their discussions of the feedback had been 
helpful and to find that the feedback included many revelations about stu- 
dents’ stated preferences. Therefore, instructors may well benefit from 
reflective discussions about their reading selections, systematic student 
feedback about readings, or both. Because course evaluation question- 
naires typically neglect reading (e.g.. Marsh & Bailey, 1993; Wilson et al., 

1 997), even when they purport to focus on an instructor’s course design 
rather than lecture perfonnance (e.g., Liow et al., 1 994), this conclusion 
points to a valuable new avenue for formative evaluation research. 

Across disciplines, the systematic feedback students provided was 
remarkably consistent. Students asked for shorter, easier, better-written, 
and more ‘relevant’ materials. The instructors I studied did not take 
these requests at face value, but instead, interpreted them as indicators of 
underlying issues of instructor-student communication, students’ moti- 
vations, and students’ cognitive abilities. Accordingly, they devised sev- 
eral responses to students’ requests, including giving students more 
infonnation about how to read, avoiding readings that are both long and 
complicated, ensuring that course readings are taken up in lectures or 
connected to graded work, and using the draw of familiar, personal, and 
contemporary topics to introduce complex analytic methods. 

An instructor perusing this manuscript for its practical recommenda- 
tions might find many helpful, or at least benign, pointers: discuss your 
reading selection with someone, test your predictions about how the sex 
(and other identities) of authors are distributed, be aware that students 
may recall late readings least, and so forth. However, of greater socio- 
logical interest is a more abstract recommendation: that instructors rec- 
ognize that their choices of readings, students’ use of and responses to 
readings, and instructors’ revisions to reading materials and pedagogical 
strategies all entail certain assumptions about how to conceive of 
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instructors’ and students’ roles. For example, if a student is thought of 
primarily as a “consumer” and an instructor as a “provider of a product,” 
in a consumer-driven market, course readings would be revised simply 
to match students’ tastes. As Child and Williams (1996) argue, such a 
model discourages critical thinking. 

The academy is without consensus on this point. Throughout this 
manuscript alone, the research participants and cited literature cast stu- 
dents and instructors in complex and sometimes contradictory roles. 
Either party may be seen as a potential victim or promulgator of a chilly 
climate. A student may be depicted as, at once, mercenary, egocentric, 
overwhelmed, independently knowledgeable, and wary of providing 
feedback. An instructor may be, at once, harried, eager to challenge, iso- 
lated by colleagues, embattled by consumerism, or surprised by the diffi- 
culty of predicting students’ preferences. 

This recommendation differs from the others in a second way. It is 
not directed to an instructor taking individual initiative to “talk to some- 
one about my readings,” so much as to instructors collectively who 
could engage in a larger conversation that identifies how political and 
institutional factors — such as government funding decisions, enrolment 
rates, tenure procedures, or methods of allocating merit pay — influence 
how the instructor-student relationship is conceptualized and the result- 
ing pedagogical practices. As Melrose (1998) has argued: 

if education is to be conceived of as emancipatory or trans- 
formative, instructors must be empowered by their organi- 
sation and by those external to it to act to continually 
change and improve teaching and learning processes for 
their students... .If the employer or the industry as customer 
or consumer is the only determinant of curriculum goals 
then teachers themselves are in danger of being disempow- 
ered to negotiate about the curriculum as professional 
educators, (p. 42)i|r 
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Notes 


' Response rates are approximate because 1 did not record the number of 
registered students in each course, but instead used the maximum possible 
enrolment in the denominator of the estimate. 

^ These scores were calculated by assigning 2 points if a student recom- 
mended keeping a reading, I point if a student had no opinion, and 0 points if 
a students recommended dropping a reading, and then taking the average of 
those points. Students who did not remember a reading were omitted from 
this calculation. 

^ One “Humanities” course is omitted here because the sample of only 
seven readings is unsuited to so detailed an analysis 

^ Because of ties among ratings, the groups of readings are not precisely of 
the desired sizes. 

^ Although my sample of instructors found that readings taken up in class 
were well-rated. Bean (1996) recommends making students responsible for 
material not covered in class to break “the vicious reading cycle. ..teachers 
explain readings in class because students are poor readers; students read poorly 
because teachers explain the readings in class” (p. 139). 
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